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ABSTRACT 

This report describes and evaluates the initial data 
collection system of the Department of Labor's Job Training 
Partnership Act (JTPA) and focuses on recently proposed revisions to 
reporting requirements. Focus is placed on the collection of data on 
the Title II program for disadvantaged youths and adults. After 
describing the system generally, the report outlines specific 
limitations of each collection method used, including the job 
training annual and quarterly status reports and the separate 
components of the longitudinal survey, including the Quick Turnaround 
Survey, Longitudinal Participant Survey, and the Longitudinal 
Comparison Survey (never implemented). Next, recent developments are 
discussed, and suggestions for additional revisions designed to 
eliminate the shortcomings of the present system are presented. The 
revised system should provide better data for use in setting 
performance standards and considerably more detailed information for 
program management and oversight. However, three particular problems 
will remain; (1) a lack of specificity of definitions; (2) 
inconsistency in the data items maintained among service delivery 
areas; and (3) lack of sufficient information to adequately measure 
the amount of training provided to program participants. (KH) 
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The Honorable Matthew G. Martinez 

Chairman^ Subcommittee on Employment Opportunities 
Committee on Education and Labor 
House of Representat*^ ves 

Dear Mr. Chairman: 

During the Job Training Partnership Act (JTPA) oversight 
hearings conducted by your Subcommittee on May 2, 1985/ we agreed 
to provide information on the JTPA data collection system. This 
briefing report is in response to that agreement. 

In carrying out our work^ we focused on obtaining information 
on the adequacy of the Department of Labor's JTPA data collection 
system/ Labor's January 1986 proposed revisions to its reporting 
requirements/ and the extent to which these revisions would address 
data system shortcomings. We concentrated primarily on Labor's 
system as it relates to the collection of data on the title IIA 
program for disadvantaged youths and adultS/ the largest program 
authorized under JTPA/ in terms of both the number of participants 
and program funding. ^ 

Our work was done in Washington/ D.C./ where we spoke with 
officials from the Department of Labor responsible for JTPA ac- 
tivities and representatives from eight groups in the job training 
community — the National Alliance of Business/ tho National Asso- 
ciation of Private Industry Councils/ the National Commission for 
Employment Policy/ the National Governors' Association/ the Na- 
tional Conference of State Legislatures/ the National Job Training 
Partnership/ Inc./ the National Association of Counties/ and the 
U.S. Conference of Mayors. We also spoke by telephone with offi- 
cials from 13 states and two service delivery areas (SDAs) — units 
of local government that carry out job training services. In 
addition/ we chaired a panel discussion by representatives of the 
job training community to discuss JTPA data collection issues. 
Most of our work was done from July through November 1985. We 



The act also authorizes a summer youth program (title IIB); a pro- 
gram for dislocated workers/ e.g./ those who have lost their jobs 
because of plant closings (title III); and federally administered 
programs/ e.g./ Job Corps and programs for Native Americans (title 
IV). (Titles I and V address general program and administrative 
issues and contain changes to other federal programs/ e.g., state 
employment service agencies.) 



ERIC 



3 



B-221 143 



delayed completing this report to review Labor's January 1986 pro- 
posed revisions, but we did not solicit the views of members of the 
job training community on the proposed revisions. 

In summary, we noted that although Labor's existing data 
collection system has limitations, its recently proposed reporting 
revisions, if approved by the Office of Management and Budget, 
should eliminate many of these shortcomings. Labor has made sig- 
nificant progress since its revised system should provide better 
data for use in setting performance standards and considerably more 
detailed information for program management and oversight. 

However, because some limitations will remain, important pro- 
grammatic data useful for congressional oversight may continue to 
be missing or may be unreliable. Chief among these limitations are 
(1) the lack of specificity of definitions, which will result in 
data being accumulated that are not similar and therefore cannot be 
reliably interpreted or compared across states and SDAsy (2) in- 
consistency in the amount and type of data maintained among SDAs, 
which may result in national estimates of JTPA participant charac- 
teristics and termination outcomes that are not representative; and 
(3) lack of sufficient information to adequately measure the amount 
of training provided to program participants. (These limitations 
restrict the usefulness of Labor's current source of data on in- 
dividual enrollees and terminees — the data set needed for analysis 
of participant characteristics in relation to services provided and 
program outcomes.) Our observations on these limitations are sum- 
marized beginning on page 18. 

Labor's new plan for evaluating JTPA through a series of 
experiments in up to 20 SDAs is a significant improvement over La- 
bor's original approach, although it too may have some limitations 
(including such practical problems as gaining the cooperation of 
program operators in conducting the experiments). However, no 
evaluation approach is totally free of problems, and Labor's pro- 
posed evaluation is a superior means of measuring program effect. 

As agreed with your office, we did not obtain official com- 
ments from Labor. However, we gave Labor officials an opportunity 
to review a draft of this document, and their comments have been 
included where appropriate . 

We are sending copies of this briefing report to the Secretary 
of Labor, the Office of Management and Budget, and other interested 
parties and will make copies available to ethers upon request. 
Should you need additional information on the contents of this 
document, please call me on 275-5365. 



Sincerely yours, 




William Ju Gainer 
Associate Director 
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JOB TRAINING PARTNERSHIP ACT; 



DATA COLLECTION EFFORTS AND NEEDS 

BACKGROUND 

On October 1, 1983, the Job Training Partnership Act (JTPA) 
replaced the Comprehensive Employment and Training Act (CETA) as 
the nation's primary federally funded employment and training 
program. Under JTPA, many administrative and oversight func- 
tions have been shifted from the Department of Labor, the fed- 
eral administ^=^r ing agency, to the states. Under JTPA the gover- 
nors are responsible for overseeing the performance of local 
service delivery areas (SDAs) through which job training serv- 
ices are provided. SDAs may include the entire state or one or 
more units of local government. JTPA title IIA, the training 
program for disadvantaged adults and youths, has been funded at 
about $1.9 billion annually through June 30, 1986. 

JTPA is a performance-oriented program. The act states 
that the basic measure of performance is the increase in employ- 
ment and earnings and the decrease in welfare dependency. To 
determine if these objectives are met, the act requires the 
Department of Labor to establish performance standards. The 
Secretary of Labor is given wide latitude to set standards and 
to specify outcomes, such as job placement and improved employ- 
ability of youths, as well as longer term outcomes that relate 
to increased employment and earnings and decreased welfare 
dependency. 

In carrying out these requirements. Labor is authorized by 
the act to direct the states to maintain and report specified 
information on program performance. The act also directs Labor 
to provide program oversight, to evaluate the effect of JTPA on 
participant earnings and welfare status, and to submit an annual 
report to the Congress summarizing the findings on and achieve- 
ments of JTPA. To comply with these provisions. Labor estab- 
lished a data collection system, of which some portions are 
still in existence, others are being revised, and still others 
have been or are to be discontinued. The following describes 
Labor* s data collection efforts. 

LABOR* S INITIAL JTPA 
DATA COLLECTION SYSTEM 

In implementing JTPA, Labor established national perform- 
ance standards for measuring participant outcome at termination 
based on experience from the prior CETA program. In addition, 
it intended to establish standards for measuring JTPA's impact 
on participants' postprogram labor-market experience and welfare 
status using data accumulated from a proposed comprehensive re- 
porting system. In April 1983, after consulting with state and 



local officials, congressional staff members, and other repre- 
sentatives of the employment and training community, Labor pro- 
posed a system that would have required states to collect and 
report data from each SDA on title IIA participants' preprogram 
and postprogram economic experience, Postprogram data were to 
be collected through follow-up contacts with former participants 
13 weeks after teruination. 

In July 1983, the Office of Management and Budget (0MB) 
rejected Labor's proposal, stating that such universal reporting 
requirements were not needed to meet federal responsibilities, 
that sampling could be used instead, that postprogram data col- 
lection was too burdensome to states and SDAs, and that Labor's 
proposed long-term study (the Job Training Longitudinal Survey) 
for evaluating program effectiveness could provide some of the 
information needed to establish national standards. Labor, 
therefore, dropped its proposed postprogram reporting require- 
ment. 

The resulting JTPA annual status report, approved by 0MB in 
September 1983, contained considerably fewer data items than 
originally proposed on terminees' socioeconomic characteristics 
and no information on participant postprogram follow-up. The 
annual status report, as approved by 0MB, gave Labor summary 
data on each SDA's program participants and terminees, the ter- 
minees' socioeconomic characteristics, and program performance 
in relation to the standards for measuring immediate outcomes. 
Its primary purpose has been to develop performance standards at 
the national level and give the states information to adjust the 
standards to take into account local economic, geographic, and 
demographic factors. As noted, follow-up information on the 
performance of participants after leaving the program was elimi- 
nated. In addition, the level of detail was reduced for many 
categories, including youth employabil ity enhancement termina- 
tions, age, education, family status, unemployment compensation, 
labor-force status, and wage and welfare data. 

Labor also proposed, and in September 1983 0MB approved, a 
JTPA quarterly status report from the states to obtain summary 
data on the number of participants and program costs. This re- 
port provided statewide program expenditure data. While the 
states were required to submit this report each quarter during 
the first 9 months ot JTPA, they now only have to do so an- 
nually • 

Labor also looked to its Job Training Longitudinal Survey 
as another source of program data. This survey was designed to 
collect data on a national sample of JTPA participants and three 
comparison groups of nonparticipants. As originally planned, 
the survey consisted of (1) a quick turnaround component, (2) a 
longitudinal component for JTPA participants, and (3) a longi- 
tudinal component for nonparticipant comparison groups. 



The quick turnaround component was designed to gather de- 
scriptive statistics on a sample of JTPA enrollees and terminees 
on a quarterly basis from a sample of 194 of the 594 SDAs na- 
tionwide. The major objective of this component was to permit 
national estimates of the nun.bers and characteristics of en- 
rollees and terminees as well as termination outcomes of major 
program activities within 4 months after each quarter. 

A more detailed, longitudinal component was to provide 
extensive data on the socioeconomic characteristics and labor- 
force experiences of a subsample of the enrollees included in 
the quick turnaround sample. This component was to merge the 
limited quick turnaround data with information obtained in three 
personal and telephone interviews, conducted about 6, 12, and 
21 months after entry into a JTPA program. These detailed data 
would then document the socioeconomic and labor- force histories 
of JTPA participants both before and after program participation 
(as well as in-program experiences). These data would also 
permit the examination of gross preprogram to postprogram 
changes in earnings and labor-force participation and the 
analysis of net program impact natioriwide. 

A comparison group component was to provide longitudinal 
data on three sets of nonparticipants. These files were to be 
used in conjunction with the file of JTPA participants, along 
with earnings data, to perform the impact evaluations. The com- 
parison groups were to be developed from two existing Census 
Bureau surveys — the Current Population Survey and the Survey of 
Income and Program Participation. A third survey, the Survey of 
History of Work, was to be developed expressly for JTPA. 

LABOR'S INITIAL SYSTEM 
HAD LIMITATIONS 

The initial system for obtaining information on JTPA pro- 
grams through the various reports and the longitudinal survey 
had limitations; as a result, the employment and training com- 
munity raised questions about the validity and reliability of 
the data that had or would have been provided. Specific limita- 
tions associated with each data collection method are discussed 
below. However, one problem common to all collection methods is 
the lack of specificity of definitions. 

Labor has interpreted its role under JTPA to be one of pro- 
viding broad policy guidance and oversight; therefore, it has 
implemented regulations that give the states authority to inter- 
pret most provisions of the law. Consistent with that view. 
Labor has provided few specific definitions of program terms. 
Although it has provided a glossary of definitions, such terms 
are broadly defined and allow the states significant discretion 
in determining what a particular term covers. 
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For example, Labor's definition of "job placement" would 
permit one entity to define a placement as being accepted for 
employment regardless of whether it was only for a short time or 
on a part-time basis. Another may require that a person be em- 
ployed for a specified number of weeks and work a specified 
number of hours per week before being considered a placement. 
As another example, the way states define a participant's 
"termination" from the program may vary. A termination may 
occur in one state immediately after a person completes the 
training program. Another state may place a person in a "hold- 
ing status" for a specified period following completion of 
training and only then is he or she categorized as terminated. 
As a result of these different definitions, the data collected 
will lack consistency and confuse interpretation of program 
results.^ This raises the question as to whether the data can 
be reliably interpreted or compared soundly across states and 
SDAs. 

The job training community and state and local officials we 
spoke with unanimously agreed on the need for more specificity 
of terms within the JTPA data system. For the most part, they 
believed these terms should be limited in number but should in- 
clude items used in comparisons. Three state officials and rep- 
resentatives of two job training community groups we spoke with 
suggested that more specific terms could be developed by an 
advisory group similar to one used to develop performance stand- 
ards. 

The following is a discussion of other limitations of each 
of Labor's data collection methods. 

Job training annual 
status report 

The annual status report contains summary information on 
JTPA participation and terminations collected annually by each 
of the 594 SDAs and forwarded to Labor through the states. It 
also contains data on performance and socioeconomic measures for 
terminees. As discussed, the report Labor initially proposed to 
0MB contained many more data items than the report that was 
eventually approved. As a result, information that was origin- 
ally considered important by Labor was not available from the 
local level. For example, 

— there are no postprogram follow-up data; 

— age distinctions are limited, especially with respect to 
youths and young adults; 



Ipor example, the different time frames for terminations may 
make suspect any cross-state analysis of length of program 
stay. 
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— the severity of unemployment, in terms of length, is not 
identified; 

— the length and type of welfare is not identified; and 

— educational attainment beyond high school is not recog- 
nized . 

Such information would have given Labor better data to use in 
setting performance standards and more descriptive information 
on who is being served by JTPA programs. 

Members of the employment and training community have 
pointed out that because the annual status report, as now de- 
signed, does not include all outcomes permitted by JTPA, it may 
direct programs away from the more disadvantaged clients. In 
particular, youth employment competency attainment (i.e., 
employment-enhancing skills, such as proficiency in basic educa- 
tion skills) is not reported separately. As indicated by Labor, 
this could result in program administrators focusing on activi- 
ties that result in placements rather than on developing skills 
that would improve the employability of disadvantaged in-school 
youths. This could mean severe restrictions in programs aimed 
at dropout prevention. The employment and training community 
also questioned whether requiring only annual reporting gave 
Labor timely enough information to adequately manage JTPA. 

Job training quarterly 
status report 

This report was submitted by the states to Labor quarterly 
during the first 9 months of JTPA and annually thereafter. It 
contains statewide summary data on the total number of partici- 
pants and total expenditures for training, participant support, 
and administration. 

The expenditures information included in the quarterly 
status report on the states' set-aside programs^ cannot be 
separately identified, nor can the number of participants served 
by each such program be determined. As a result. Labor has in- 
complete information on the use of these funds and is unable to 
respond routinely to congressional and other requests for set- 
aside information. Moreover, no program-wide information on 



2Twenty-two percent of a state's JTPA title IIA funding is set 
ajide for special purposes. Specifically, 5 percent is set 
aside for state administrative costs, 6 percent for incentive 
grants and technical assistance, 8 percent for state education 
coordination and grants, and 3 percent for state older worker 
programs . 
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funding and participants served under the set-aside provisions 
is available from other data sources. 

Members of the employment and training community question 
whether the OMB-approved annual reporting requirement is of suf- 
ficient frequency. Labor officials believe that more frequent 
reporting would be desirable, as discussed on page 17. 

Job Training Longitudinal Survey 

Labor's long-term effort to measure the impact of JTPA con- 
sists of three segments — a quick turnaround component and two 
longitudinal components- Each has shortcomings that raise ques- 
tions about the usefulness and reliability of the data col- 
lected. 

Quick turnaround component 

The quick turnaround data, which are drawn from a nation- 
wide sample of 194 SDAs, are used to provide national estimates 
of JTPA enrollee and terminee characteristics as well as termi- 
nation outcomes. These data supplement the annual status report 
submitted by each SDA by providing more detailed descriptive 
statistics on enrollees and terminees and do so on a quarterly 
basis. Therefore, this report is available much earlier than 
the annual status report. The quick turnaround survey data, 
however, were intended to provide nationwide estimates and can- 
not be used to furnish data on individual state or local pro- 
grams. In addition, although the survey attempts to obtain 
information on length and type of training provided, it does not 
seek data on the quantity of training in hours — a major indica- 
tor of the extent to which services are received. Further, 
while the survey seeks more data items than required by the 
annual status report, its source is the SDA records, so the 
availability of such data depends on whether they are collected 
and recorded by the SDAs. 

Many SDAs collect data above and beyond the limited amount 
required for the annual report; however, they do not do so con- 
sistently. An analysis by a private contractor (SRI Interna- 
tional) hired by Labor showed that, in fiscal year 1984, over 
one-third of the SDAs did not maintain certain participant 
characteristic data collected through the quick turnaround 
survey. Such data included the number and age of dependents, 
whether individuals had exhausted the^r unemployment compensa- 
tion benefits, and the average wage on their previous job. 
According to Labor, data items with a nonresponse rate greater 
than 10 percent are not used in its analysis. As a result, some 
data items requested in the quick turnaround survey are not 
available for such analysis. The contractor also noted that the 
time frame for certain items varied. For example, while about 
90 percent of the SDAs recorded preenrollment work history 
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information, some obtained this information on participants for 
the 13 weeks prior to program enrollment, and others recorded it 
for the prior 26 weeks* 

As discussed by a representative of the job training com- 
munity, if an agency does not collect a large proportion of the 
information being sought, the quicV. turnaround results may not 
accurately reflect the experience of the participants in the 
<=?ample and, therefore, may not be representative. Furthermore, 
even if collected by most SDAs, the data still may lose some 
value because the information may not have been gathered system- 
atically and consistently. In addition, as noted, the data en- 
able only nationwide estimates and have limited or no validity 
for any particular state or locality. Also, since the quick 
turnaround data do not include quantity of training in hours, 
the extent of training provided to program participants cannot 
be adequately measured. For example, there is no way of knowing 
whether welfare recipients who were in classroom training for 10 
weeks received 40 hours of training or 400. 

The quick turnaround data base (including the longitudinal 
participant subsample) is the only current source available con- 
taining socioeconomic characteristics of individual partici- 
pants. As a result, it is the only data source permitting 
analyses of multiple characteristics, including multiple charac- 
teristics in relation to services provided and program outcomes 
(i.e., cross-tabulations). Such detailed data enable refined 
analyses of who is being served. For example, the age and edu- 
cation level, sex, race, and welfare status of those receiving 
on-the-job training might be determined. The other sources con- 
tain only aggregate data and therefore — unlike individual data 
bases — cannot be used to determine who was served in relation to 
services received and outcomes. 

Analyses of individual data are important for program over- 
sight. They also could be helpful in establishing performance 
standards and in developing a methodology for adjusting such 
standards to reflect local conditions. 

job Training Longitudinal 
Survey components 

Certain problems associated with the longitudinal component 
data limit their usefulness for evaluating the net impact of 
JTPA. These problems can be categorized as data base limita- 
tions and methodological constraints. 

Data base limitations include limited availability of earn- 
ings data for participant and comparison groups because of con- 
fidentiality concerns and limited sample sizes of participant 
and nonparticipant groups. The JTPA long-term survey, which was 
modeled after CETA^s longitudinal survey, would face the same 
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methodological constraints as those experienced under that pre- 
vious survey, particularly in the construction of comparison 
groups. This would preclude adequately measuring the effects of 
JTPA on participants' labor market success* 

LABOR'S DATA COLLECTION FRAMEWORK 

Table 1 summarizes Labor's initial JTPA data collection 
efforts, including a description of ^ach, its purpose, and its 
limitations. 

Table I; 

Labor *s TnltiaT Data Collection Framework for JTPA 



Rqorts 

ikmual Status 
Report 



Source of data 

All SG^ thnx;^ 
out the state 



Description 

SDArwide, cumilative 
data on total partici- 
pants, termination 
status, performance 
nBasures, and socio- 
economic characteris- 
tics of all terminees. 



Purpose 

To obtain 
information 
on terminee 
character- 
istics. 



Limitations 

Includes relatively 
few socioeconomic 
data iteoQS and no 
youth employment 
axageteacy outcomes. 
Data items aggregated, 
oaakipg analysis diffi- 
cult. Annual report- 
ixig only. Inconsis- 
tency between state 
definitions. 



Quarterly 
Status 
I^rt 
(E>qpeniiture 
Report) 



All states on a 
state-vLde basis 



Sunnary cumulative data 
on participation and 
progran costs. 



To determine 
nuDbers served 
and expendi- 
tures* 



Submitted annually. 
Inconsistency between 
state definitions. 



Job Training 
Loq^tudinal 
Survey 

Quick 
Tijmaround 
Survey 



Quarterly sauple 
of 6,000 en- 
rollees and 3,000 
terrainees by Cen- 
sus Bureau f torn 
a saaq>le of 194 
SDAs 



DeaiogrqMc, socio- 
economic, and program 
participation data 
obtained from admLnis- 
trative files. 



To provide 
descriptive 
statistics on 
enrollees and 
terminees and 
to supplement 
the annual 
status report. 



Provides nationwide 
data only. No geo- 
graphical breakout. 
Inconsisteit data be- 
tween SDAs. Inconsis- 
tent recordkeeping by 
SDAs results in Low 
response rates for 
some items. No data 
available on quantity 
of trainii>g by hours. 



BEST COPY AVAILABLE 
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Reports 



Source of data Description 



Purpose 



T limitations 



Longitudinal 
Participant 
Survey 



Loi^gltudinal 
Qcxiparison. 
Survey 
(never 

iiq)leQ£nted) 



An annua], sub- 
sample of 10,000 
enrollees from 
the quick turn- 
around saople 



TU> ongoing 
Census Bureau 
surveys and 
another created 
for JTPA 



Provides micro-^ta on 
participants, including 
preprogram deapgraphlc 
and socioeconomic data, 
program participation 
data, and 21-month 
f oUow-tq) data obtained 
throu^ personal inters 
views with participants. 

Provides micro-data on 
control group, indudr 
ing deDaQgra{)hic and 
socioeconomic data over 
various time periods 
obtained throv^ per- 
sonal interviews with 
noqpartlcipant control 
group. 



To examine the 
gross pre- to 
postprogram 
changes in 
earnings and 
labor-force 
experiences 
and to assess 
program iiq;>act« 

To caq;>are 
program par- 
ticipants with 
nonpartici- 
pants to meas- 
ure impact of 
program* 



Problems with earnings 
data, geographic data, 
and timing may hanq)er 
impact evaluation* 
Data would not be 
available for several 
years because of the 
nature of longitudinal 
studies* 

bample size may be 
insufficient for soue 
subgroqp net iiq)act 
analysis* Problems 
with earnings data, 
geographic data, and 
timf T>g may hamper 
isnpact evaluation* 
Data would not be 
available for several 
years because of the 
t£iture of lopgitudinal 
studies* 



RECENT DEVELOPMENTS IN LABOR *S 
JTPA DATA COLLECTION EFFORTS 

In January 1986, Labor initiated actions to change its data 
collection system* It plans to phase out the longitudinal par- 
ticipant component of its long-term study, eliminate the longi- 
tudinal comparison groups, and modify the quick turnaround com- 
ponent* It also has proposed, for the first time since the 
inception of JTPA, significant changes to its annual and quar- 
terly status reports* 



Portions of long-term study 
to be eliminated or modified 

Because of problems recognized in its longitudinal study of 
JTPA, some of which were discussed earlier. Labor has undertaken 
a number of corrective actions* It had several consultants 
evaluate specific aspects of its JTPA evaluation plans* It also 
formed a technical advisory panel made up of recognized experts 
in social research to examine the plans for the Job Training 
Longitudinal Survey* Based on the panel's report that Labor's 
longitudinal study could not determine net impact. Labor devel- 
oped a new plan for evaluating JTPA programs* 
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Labor's plan consists of several initiatives. Most nota- 
bly^ in place of a national comparison group to estimate program 
impact^ it intends to undertake field experiments at up to 20 
SDAs to evaluate the benefits and costs of JTPA services for 
various target groups. The experiments will involve randomly 
assigning JTPA-eligible persons who apply to the program to two 
groups — one receiving JTPA services and the other ^ a control 
group y not receiving services. The random assignment approach 
is aimed at eliminating the major problem with constructed 
comparison groups — that JTPA participants are compared with non- 
participants who may not be similar with respect to important 
characteristics. Such differences^ for example^ can bias esti- 
mates of changes in earnings. Labor believes that within 
5 years it will have net impact estimates that will (1) assert 
cause and effect^ (2) rely less on sophisticated statistical 
modeling and thus produce findings that are less open to ques- 
tion and (3) command a broad professional consensus since the 
data would yield a clear set of results. 

Labor recognizes that using random assignment will not be 
popular with many program operators and applicants; it may re- 
sult in turning away some needy applicants. (This has created 
problems in other evaluations using this approach.) However r 
because JTPA programs face a large oversupply of eligible appli- 
cants ^ Labor believes random assignment to be fair. Other prac- 
tical problems of implementation also may arise ^ and there may 
be questions as to the generalizability of the results since 
they will be from selected sites rather than from representative 
national samples. Further^ as with any long-term evaluation^ 
timeliness may be a concern. 

According to Labor^ the above initiative will permit it to 
phase out the longitudinal components of its Job Training Longi- 
tudinal Survey. This is being done on a staggered basis^ begin- 
ning in January 1986. In the interim^ Labor will continue to 
gather data on transition year (Oct. 1^ 1983-June 30 ^ 1984)^ and 
program year 1984 (July 1^ 1984-June 30^ 1985) enrollees for 21 
months after program entry. In addition ^ since Labor will no 
longer rely on its longitudinal study for net impact estimation ^ 
it will reduce the quick turnaround enrollee sample from 24^000 
to 12^000 enrollees annually.^ 

Labor also is planning a 12-month follow-up on a small 
sample of terminees (1^000) from the quick turnaround sample 
which will be designed to gather information on participants' 
postprogram experiences. Labor hopes to be able to calculate 
employment rates ^ earnings^ and welfare dependency rates for 
males and females and for adults and youths. However , because 



3of the 12^000 enrollees^ 8^400 are in title IIA programs^ and 
the remainder are in title III. 
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of the small sample size^ data may not be available by type of 
service or for other subpopulations. Labor is reviewing the 
feasibility and cost effectiveness of this approach. 

New proposal for reporting requirements 

In January 1986^ for the first time since the inception of 
JTPAr Labor proposed significant revisions to its annual status 
report in order to extend and update its reporting system. 
These new reporting proposals were forwarded to 0MB for review 
and approval on January 10^ 1986. 0MB was still reviewing the 
proposals as of March 25 ^ 1986. The reasons for the revisions 
cited by Labor to 0MB include: 

— Labor anticipates adding new performance standards for 
program year 1988 to measure postprogram outcomes and 
youth competency attainment. Therefore ^ d ita collection 
must begin in program year 1986 in order to obtain the 
information on which to base the standards. 

— The revisions will permit identifying more adequately 
those who are difficult to serve and adjusting perform- 
ance standards when SDAs serve a disproportionate number 
of such individuals. 

Among the revisions of the annual status report are addi- 
tions to data collected on program performance ^ participant 
characteristics^ and program eligibility. Also^ youth employ- 
ment competency attainment will become a part of this reporting 
system* Significant among these changes is the proposed collec 
tion of postprogram data through participant follow-ups. In it 
justification for that addition ^ Labor stated that its proposed 
postprogram measures: 

"... will focus program operators on the employment 
and earnings and job retention of participants three 
months after termination. This focus will signifi- 
cantly strengthen the quality of the program over the 
current termination-based approach to program design 
and service delivery which merely emphasizes getting 
participants jobs." 

Labor stated that failure to add postprogram follow-ups 

"... would raise questions as to whether it was con- 
forming to the act; and continue to focus program de- 
sign and service delivery on placement rather than on 
the goals of JTPA — to increase (long-term) earnings 
and reduce welfare dependency of those served by the 
program. " 
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Labor also proposed (1) revisions and additions to its 
quarterly status report on expenditures and (2) the initiation 
of a new annual report on participants in the summer youth pro- 
gram. In its justification to 0MB for the proposed revisions to 
the quarterly report^ Labor stated that it currently has incom- 
plete information on how states are using the set-aside funds 
provided under JTPA^ that this report does not measure state 
performance against the requirement that 40 percent of title IIA 
funds be spent on youths^ and that little management information 
exists to evaluate the performance of title III national 
reserve- funded programs or to develop options for decision 
makers. Labor is also proposing that this report be submitted 
semiannually in order to respond to congressional inquiries 
during the budget process and to provide a sound basis for its 
budget recommendations. 

Labor is proposing the new report on the summer youth pro- 
gram because it now has no information available to measure who 
is being served under this portion of the act and to respond to 
congressional inquiries. This report would provide statewide 
participant characteristics data. 

IMPROVEMENTS IN THE NEW REPORTING 
PROPOSALS AND SUGGESTED ADDITIONS 

The new reporting requirements proposed by Labor ^ if ap- 
proved by 0MB r should address many of the problems and short- 
comings noted by those in the job training community. These 
requirements should increase the data available for setting 
performance standards and for program management and oversight 
and result in more consistent data. Also^ members of the job 
training community that we spoke with generally indicated that 
they would not be averse to additional reporting requirements. 
Some members of that community have suggested certain additional 
requirements. 

Improvements in data 
collection system 



Because Labor's new reporting requirements were only re- 
cently submitted to 0MB for review^ we did not have the time to 
solicit the views of members of the job training community on 
the specific proposals. However ^ these proposals should address 
many of the limitations the community found in Labor's current 
data collection system. 

Annual report revision 

The revised annual status report includes many of the data 
items included in Labor's initial proposal to 0MB. Chief among 
these is the collection of information on the postprogram 
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labor-market experiences of JTPA terminees. According to Labor, 
postprogram measures are superior to termination-based measures 
in evaluating program performance. 

The annual report revisions will also provide additional 
terminee characteristics information which, according to Labor, 
will permit it to better determine whether JTPA programs are 
serving those most difficult to serve* This, in turn, will pro- 
vide the information necessary to adjust performance standards 
when SDAs disproportionately serve such persons. Labor stated 
that all the additional characteristics data items have been 
shown in statistical analyses to be important determinants of 
performance . 

It is proposed that youth employment competency attainment 
be reported in detail. By including this element in its annual 
report. Labor will increase the visibility of this outcome and 
may encourage SDAs to serve youth dropouts and potential drop- 
outs, and lay the groundwork for youth employment competency 
performance standards. However, as noted with other terms, 
youth employment competency will need to be more explicitly 
defined. 

Lastly, the annual report will give Labor information on 
the eligibility of those being served. Under JTPA, at least 90 
percent of the participants must be economically disadvantaged. 

Other report revisions 

The revisions to the quarterly expenditure report and the 
addition of a report on summer youth employment programs will 
give Labor additional information to manage and monitor JTPA. 
As Labor pointed out, it has incomplete information on state use 
of set-aside funds, no information on levels of participation, 
and no information on the characteristics of summer youth em- 
ployment program participants. The revisions would correct 
these deficiencies, but there would be no information on client 
characteristics and outcomes from state set-aside funded pro- 
grams. The revisions would also permit the semiannual measure- 
ment of state performance against the 40-percent youth expendi- 
ture requirement of JTPA. Reporting these data semiannually, as 
proposed, would enable Labor to respond to congressional in- 
quiries during the budget process and provide a better basis for 
its budget recommendations. Finally, adding a new summer youth 
employment report would give Labor information on who is being 
served under this program. Currently, no participant character- 
istics data are reported for this program. 
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Proposed reporting revisions 



not viewed as burdensome by Labor 



As discussed, 0MB rejected Labor's original data collection 
proposal as being too burdensome on the states. In its January 
1986 proposal, Labor stated that the revised requirements for 
the annual status report are not expected to increase the re- 
porting burden for the states and SDAs, except for the post- 
program data collection, which will involve new procedures. 
Labor estimated that nationally the cost of follow-up will total 
about $3 million annually. Labor also noted that "federal 
reporting is the most effective method for collecting program 
performance and participant characteristics." 

Labor also stated that the revised quarterly status report 
and the new summer youth report should not increase most states' 
reporting burden. Current systems designed to respond to pre- 
sent requirements aggregate the participant and expenditure data 
collected by SDAs on formula and set-aside funds. Labor stated 
that, while existing systems in some states may require modifi- 
cation to produce additional participant and expenditure detail 
called for in the revised report, the cost of modifying these 
systems should be moderate for most states. Also, a change to 
semiannual statewide cumulative reporting should not require 
system adjustments. 

Finally, according to Labor, states would not be expected 
to incur substantial costs in compiling summer youth character- 
istics data. Many states already maintain such participant in- 
formation on a centralized basis as part of eligibility determi- 
nation systems and state-SDA reporting systems. 

Others do not believe data 
collection to be burdensome 

Officials from eight job training community groups, includ- 
ing those representing state agencies, said that the states and 
SDAs did not consider data collection or reporting to be burden- 
some. Also, officials from 10 states, two SDAs, and eight job 
training community groups pointed out the advantages of having 
detailed program information. They cited the need for such in- 
formation to plan, manage, and evaluate JTPA programs. Offi- 
cials from two states said that accurate and timely data would 
help to market the successes of JTPA and help to keep the pri- 
vate sector involved. 

We contacted by telephone five states with participant 
follow-up systems in place. Estimates on the cost per follow-up 
by officials in these states varied, depending on the amount of 
data they gathered and the program activities they included. 
Their estimates ranged from $8 to $28 per contact and averaged 
about $17. Labor estimated that the cost to meet its follow-up 



ERLC 



17 

19 



requirements would be about $15 per contact • In our 1985 report 
to the Congress on JTPA implementation, 4 pointed out that 48 
states planned some follow-up on former participants to collect 
postprogram data. 

Additional report data desirable , 
according to some 

While the revisions to Labor's data collection system will 
address many of the problems and shortcomings noted, members of 
the job training community believe it would be desirable to have 
additional data items in the annual status report. For example, 
as pointed out by one member of the community, there is a need 
for information on what training is being received by those ob- 
taining employment and whether participants are being employed 
in the field for which they were trained. The other members of 
the community we spoke with agreed that such information would 
(1) help program administrators and others determine whether the 
training received was sufficient to facilitate employment and 
whether it was being offered in occupations for which openings 
existed and (2) encourage program planners to focus on those 
training services that enhance locating, obtaining, and retain- 
ing employment. 

Members in the job training community we spoke with cited 
one or more of the following data ite^s as being useful for 
refining performance standards and for other information pur- 
poses. These included: characteristics of persons served 
through the use of JTPA state set-aside funds and termination 
outcomes, persons age 55 and older, 5 veteran status, displaced 
homemakers, age distinctions of dependent children of single 
parents, and follow-up information on whether individuals remain 
on or receive reduced levels of welfare. In addition, members 
of the community recommended that the annual status report be 
made available more frequently to improve the timeliness of the 
data available for each state and SDA. 

GAP OBSERVATIONS 

In our view, the reporting proposals submitted by Labor to 
0MB in January 1986, if approved, should eliminate many of the 
shortcomings and limitations of the current data collection 
system. However, the following data limitations will continue 
to exist: 



4 jQb Training partnership Act: Initial Implementation of Pro- 
gram for Disadvantaged Youth and Adults , aRD-85-4, Mar. 4, 
1985. 

^The current annual status report contains this information, but 
Labor's proposed revisions call for "age 30 and over." 



— Lack of specificity of definitions. 



— Inconsistency in the data items maintained among SDAs. 

— Lack of sufficient information to adequately measure the 
amount of training provided to program participants. 

These shortcomings will limit information useful for congres- 
sional oversight of JTPA. 

Labor's decision to modify its long-term effort to evaluate 
JTPA because of methodological problems resulted in an approach 
that is methodologically far more sound and should be a source 
of important information on JTPA. As with any evaluation, 
however, it may have some limitations of its own. 

Labor's new reporting proposals, currently before 0MB for 
review and approval, will provide more detailed data on JTPA 
program operations. In our consultations with members of the 
employment and training community, we found wide receptivity for 
more standardized and extensive program data that would assure 
greater uniformity and comparability, and little complaint that 
this would be unduly burdensome or of little utility. 

The proposed new requirement for participant follow-up data 
is desirable. It provides valuable data on whether those re- 
ported as being placed in a job continue to hold that position 
and whether employment rates are maintained or improve as com- 
pared to earlier reported placement results. On the other hand, 
there may be technical problems — such as with follow-up methods, 
sampling, and nonresponse rates — that could limit the data's 
reliability and comparability. 

The proposed revisions to Labor's reporting requirements do 
not include any additional guidance regarding variations in the 
definitions of the terms used. The job training community and 
state and local officials we spoke with unanimously agreed on 
the need for more specificity of terms within the JTPA data 
system. 

As noted, the quiok turnaround survey seeks no information 
on quantity of training by hours. Therefore, the amount of 
training provided cannoc be adequately measured. 

The availability of consistent data for the quick turn- 
around survey will continue to be a problem. The proposed 
changes to Labor's reporting system, if approved, will require 
each SDA to collect additional data beyond that required under 
the current system. However, those additional data items col- 
lected for the quick turnaround survey but not required by 
Labor's system will, more than likely, not be consistently main- 
tained by each SDA in the sample. The quick turnaround survey 
seeks significantly more information than the data required for 
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the annual status report. For example, the survey attempts to 
obtain information on the type of training provided, whereas the 
annual report does not. In addition, the quick turnaround 
survey provides much more detailed data on reasonc for program 
termination, income, type and level of public assistance, and 
family status. 

We are uncertain how complete the quick turnaround data 
will be and whether the smaller sample cf enrollees will be as 
sufficient as the original sample for a wide array of subgroup 
distinctions. This affects Labor's only current source of in- 
dividual data on enrollees and terminees. Such individual data, 
rather than aggregated summary data, are necessary for analysis 
of groups by multiple characteristics, e.g., woiren by age and 
minorities by level of education. (The quick turnaround data 
have nationwide applicability only; analysis of participant 
characteristics, outcomes, and types of services cannot be con- 
ducted at the state or SDA level with the quick turnaround data 
source. ) 

Regarding Labor's new evaluation approach, it plans to 
undertake a "mini"-f ollow-up of 1,000 terminees from its quick 
turnaround survey. This is a timely way to obtain postprogram 
experience data, but the sample is so small that only limited 
disaggregation is possible. For example, the sample is too 
small to determine the postprogram earnings of high school drop- 
outs, the employment rates of blacks who received classroom 
training, or the reduction in welfare dependency of women ter- 
minees. It is important to note that, although the longitudinal 
participant sample is no longer needed for net impact evaluation 
and is therefore being pha.^ed out, compared to the mini- follow- 
up, it provides a much larger sample and a much richer source of 
data to*^ characterize what happened, after participation, to 
various population groups. The longitudinal participant sample 
could also have provided an extensive cross-check of Labor's 
data from administrative records. 

Labor's planned use of random assignment experiments to 
estimate program net impact is a superior means of measuring 
effect. As with any evaluation, however, there also may be 
limitations. There may be implementation problems with such a 
rigorous design and questions regarding the generali^^ability of 
the results. 

Labor has made substantial progress in redesigning its data 
collection system, which m'^st still be approved by 0MB, although 
some limitations will remain. Table 2 s-^mmarizes the improve- 
ments and limitations resulting from Labor's proposed reporting 
revisions. 
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Table 2 : 

Labor^s Proposed Reporting Rfivisions-^Inyrovecaents and Limitations 



Report 



Proposed changes 



Inprovements 



UMtations 



Annual 
Status 
Report 



Additional data itans 
on characteristics of 
program participants , 
such as "uneoiplpyed 
13 or wove weeks during 
prior 26 weeks," "post 
hi^ school education/* 
and "offender status." 
New data on program 
perfortoaixrey i«e«y 
postprogram follow-tap 
data and youth eaploy- 
ment competency attain- 
ment. 



Improved data for setting 
performance standards , 
incluiipg information on 
postprogram labor market 
experience and youth 
eaiploynent conq)etency 
attainment. Will liH)rove 
program management and 
oversight, e.g., will 
permit identifying more 
adequately those more 
difficult to serve. 



No cross-tabulations of 
data are possible since 
these are aggregated 
data. Broad definitions 
may result in data that 
cannot be reliably 
interpreted or compared 
soundly across states and 
SDAs* Some needed data 
itecDs not included, e.g., 
client characteristics 
and outcomes on the 
3-percent set-aside for 
older workers and the 
8-percent set-aside for 
education coordinatijon 
grants. Lack of time- 
liness (annual sub- 
mission). 



Quarterly Status 
Report (E»- 
penditure 
Report) 



Semiannual submission 
(currently submitted 
annually). Additional 
inf oinmation on state 
set-aside e}q)enditures, 
level of participatixxi 
under set-aside prograns, 
measurement of state 
performance against the 
40-percent youth expend- 
iture requirement. 



Enables Labor to respond to 
congressional ir-^uiries 
during the budget process 
and provides a basis for 
its budget recoaoendations. 
Alleviates scxoe weaknesses 
in labor* s knowledge of 
states* use of set-aside 
funds and allows determi- 
nation of vAiether states 
are meeting the youth ex- 
penditure requirements. 



Quarterly submission 
would be preferable in 
order to correct prob- 
lems in a timely manner. 



Suaoer Youth 
BnpLoyment 
Program 
Report 



Adds a new reporting 
requirement. Provides 
Labor with participant 
characteristics on 
those served under 
title TIB, the summer 
youth program, on a 
statewide basis. 



For the first time. Labor 
will have information on 
the cl^acteristics of 
those participating in the 
suoiner youth eaiployment 
program, e.g., age, sex, 
race/etlmicity, and educa- 
tion status. 
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Report 



Proposed changes 



ImprovfTOTits 



Limitations 



Quick 
IXxmarouod 
Surv^ 



Seduction of enroilee 
saaq)le size fran 24,000 
to 12,000* (This is the 
only source of individ- 
ual enroilee and ter- 
tuLnee data for assessing 
JTPA. While also cor^• 
taining individual data, 
the longitudinal parti- 
cipants survey will be 
phased out and the pro- 
posed mini-foHow-ip of 
terminees will consist 
of a sample of only 
1,000.) 



^klitional data items 
required by tl^ annual 
status report should result 
in a hi^^ response rate 
for individual items in 
the quick turnaround, 
since the quick turnaround 
is drawn from the same 
administrative files as 
the annual status report. 



Reduction of enroilee san 
pie from 24,000 to 12,000 
may affect analysis of 
sane subpopulations. 
Broad definitions and in- 
consistent data items may 
result in nonrepresenta- 
tive national estimates. 
Nationwide applicability 
only, i.e., cannot deter- 
mine participant charac- 
teristics, outcoaes, or 
types of services at the 
state or SLA Level. No 
data available on quantity 
of training in hours. 



Mini-follow-up 
sao^le 



Longitudinal 
Participants 
Surv^ 



New reporting require- 
ment. It is a 12-iQDnth 
follow-up involving per- 
sonal interviews with 
1,000 terminees from the 
quick turnaround. 
Gathers data primarily 
on their postprogram 
experiences, such as 
en5)loyment rates, earn- 
ings, and welfare de- 
pendency for males and 
females and for adults 
and youths. 

Labor is phasing out 
this sample beginning 
in calendar year 1986. 
It will continue to 
gather data on transi- 
tion year and program 
year 1984 airolless for 
21 ta>nths after program 
entry. 



Provides a timely means 
of obtaining postprogram 
data on participants. 



Not applicable since this 
sao^le is being phased 
out. 



Sa[q>le of 1,000 terminees 
is too small for analysis 
of some major subpopula- 
tions, e.g., postprogram 
earnings of dropouts. 



By phasing out the 
longitudinal participants 
survey. Labor will be 
eliminating a larger 
sample and richer data 
source than the raini- 
followmp vMch it is 
introducing. 
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BEST COFV AVAILABLE 



^iJlti•site field 
e3q)eriisen£s 



Proposed changes 

This approach for esti- 
matlr^ program impact 
rq>laces the lopglDjlinal 
participaiits survey and 
constructed couoparlaon 
group samples that vere 
in the original Job 
Ttalnlng Ii3ngitudinal 
Survey. Field ©qperi- 
ments will be conducted 
at 19 to 20 SDAs to 
evaluate benefits and 
costs of JTPA services 
for various target gnx^s. 
Experiments will involve 
random assignment. 



Improvements 

Because of the nature of 
the experiments, i.e., ran- 
dom assignment, the Impact 
estimates should assert 
causes and effect, rely 
less on complex statisti- 
cal modeling, and connand 
a broad professional 
consensus. 



T. imitations 

Uiere may be practical 
problems of implementa- 
tion, such as gaining the 
cooperation of program 
managers in the use of 
random assigpooent. Will 
not provide national 
estimates. Iiiierent 
problem of timeliness, 
i.e., flndli^ on Imprt 
win not be available for 
several years. 
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